What Works Clearinghouse™ U.S. DEPARTMENT OF EDUCATION 


@) WWC Intervention Report jg... 


A summary of findings from a systematic review of the evidence EDUCATION SCIENCES 


Charter Schools January 2018 


Harlem Children’s Zone (HGZ) Promise 
Academy Charter Schools 


This intervention report presents findings from a systematic review of the Harlem Children’s Zone (HCZ) 
Promise Academy Charter Schools conducted using the WWC Procedures and Standards Handbook 
(version 3.0) and the Charter Schools review protocol (version 3.0). No studies of the HCZ Promise 
Academy Charter Schools that fall within the scope of the Charter Schools review protocol meet 
What Works Clearinghouse (WWC) group design standards. Because no studies meet WWC group 
design standards at this time, the WWC is unable to draw any conclusions based on research about 
the effectiveness or ineffectiveness of the HCZ Promise Academy Charter Schools on elementary, 
middle, and high school students. Research that meets WWC design standards is needed to determine 
the effectiveness or ineffectiveness of this intervention. 


Intervention Description‘ 


The Harlem Children’s Zone (HCZ) is a non-profit organization designed to serve low-income children and families 
living in Harlem in New York City. It provides various community services and supports two public K-12 charter 
schools, hereafter referred to as the HCZ Promise Academy Charter Schools.? The HCZ Promise Academy Charter 
Schools have a longer school day and year than traditional public schools, and focus on core academic subjects, 
arts, and physical fitness. They monitor student progress on academic outcomes and provide differentiated 
instruction for students who have not met required benchmarks. The schools aim to recruit and retain high-quality 
teachers and use student achievement to evaluate and incentivize teachers. In addition to focusing on academics, 
the schools educate students and families on character development, healthy lifestyles, and leadership skills. 

In partnership with the HCZ, the HCZ Promise Academy Charter Schools provide students with various community 
services and non-academic supports, such as social workers, counseling, and medical and dental services. 


Research? 


The WWC identified three studies of the HCZ Promise Academy Charter Schools for elementary, middle, and high 
school students that were published or released between 1996 and 2017.4 


One study (Yeh, 2013) is out of the scope of the Charter Schools review protocol because it has an ineligible study design. 


Two studies are out of the scope of the Charter Schools review protocol for reasons other than study design. Both 
studies are ineligible because they examine the effectiveness of an individual charter school, but this WWC review 
focuses on the HCZ organizational model.° The HCZ organizational model is a group of two charter schools, the 
HCZ Promise Academy Charter Schools, organized as a charter network. Any effect of the HCZ model cannot be 
distinguished from the effect of the particular staff, local services, or other factors at the school that are not part of 
the HCZ model. One of these two studies (Dobbie & Fryer, 2015) is a randomized controlled trial that examined 
the effectiveness of an HCZ Promise Academy Charter School on student achievement and matriculation of 
postsecondary students. The other study (Whitehurst & Croft, 2010) is a quasi-experimental design that compared 
the New York State mathematics and English language arts achievement of students in a HCZ Promise Academy 
Charter Schoo! with that of students in charter schools in the Bronx and Manhattan between 2007 and 2009. 


Harlem Children’s Zone (HCZ) Promise Academy Charter Schools January 2018 Page 1 


WWC Intervention Report 


References 


Studies that meet WWC group design standards without reservations 
None. 


Studies that meet WWC group design standards with reservations 
None. 


Studies that do not meet WWC group design standards 
None. 


Studies that are ineligible for review using the Charter Schools Evidence Review Protocol 


Dobbie, W., & Fryer, R. G., Jr. (2015). The medium-term impacts of high-achieving charter schools. Journal of 
Political Economy, 123(5), 985-1037. The study is ineligible for review because it is out of scope of the protocol. 
Additional sources: 

Dobbie, W., & Fryer, R. G., Jr. (2009). Are high-quality schools enough to close the achievement gap? Evidence 
from a social experiment in Harlem (NBER Working Paper 15473). Cambridge, MA: National Bureau of 
Economic Research. Retrieved from https://eric.ed.gov/?id=ED507097 

Dobbie, W., & Fryer, R. G., Jr. (2011). Are high-quality schools enough to increase achievement among the 
poor? Evidence from the Harlem Children’s Zone. American Economic Journal: Applied Economics, 
3(3), 158-187. 

Dobbie, W., & Fryer, R. G., Jr. (2013). The medium-term impacts of high-achieving charter schools on non-test 
score outcomes (NBER Working Paper 19581). Cambridge, MA: National Bureau of Economic Research. 

Dobbie, W., & Fryer, R. G., Jr. (2014). The medium-term impacts of high-achieving charter schools. Retrieved 
from: https://scholar.harvard.edu/ 

Dobbie, W., & Fryer, R. G., Jr. (2014). The medium-term impacts of high-achieving charter schools on non-test 
score outcomes. Web appendices. Retrieved from: https://scholar.harvard.edu/ 

Whitehurst, G. J., & Croft, M. (2010). The Harlem Children’s Zone, Promise Neighborhoods, and the broader, 
bolder approach to education. Washington DC: Brookings Institution, Brown Center on Educational Policy. 
Retrieved from https://eric.ed.gov/?id=ED511164 The study is ineligible for review because it is out of scope 
of the protocol. 

Yeh, S. S. (2013). A re-analysis of the effects of KIPP and the Harlem Promise Academies. Teachers College Record, 
115(4), Article No. 040301. The study is ineligible for review because it does not use an eligible design. 


Harlem Children’s Zone (HCZ) Promise Academy Charter Schools January 2018 Page 2 


WWC Intervention Report 


Endnotes 


1 The descriptive information for this intervention comes from the Harlem Children’s Zone (HCZ) website at http://hcz.org, 
downloaded June 2017. The What Works Clearinghouse (WWC) requests developers to review the intervention description sections 
for accuracy from their perspective. The WWC provided the developer with the intervention description in June 2017; however, the 
WWC did not receive a response. Further verification of the accuracy of the descriptive information for this intervention is beyond 
the scope of this review. 


2 The two public K-12 charter schools operate in different sites, and each site has four schools: lower elementary, upper elementary, 
middle school, and high school. 


5 The literature search reflects documents publicly available by February 2017. Reviews of the studies in this report used the standards 
from the WWC Procedures and Standards Handbook (version 3.0) and the Charter Schools review protocol (version 3.0). The evidence 
presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 
The WWC released a quick review of Dobbie and Fryer (2009) in March 2010. The current study rating differs from the prior quick 
review rating. The quick review used the standards from the WWC Procedures and Standards Handbook (version 1.0) and the Quick 
Review protocol (version 1.0), which considered Dobbie and Fryer (2009) eligible for review of a study of the effectiveness of an 
individual charter school. The study was rated as meets standards without reservations in the 2010 review. The study was reviewed 
again under the Charter Schools review protocol for this product, and was determined to be ineligible for review. Under the Charter 
Schools review protocol, a study of the effectiveness of an individual charter school is not eligible to be included in a review for 
evidence of the effectiveness of a named Charter Management Organization (CMO) or charter network. Dobbie and Fryer (2009) was 
conducted in one of HCZ’s Promise Academy Charter Schools, so any effect of the HCZ Promise Academy Charter Schools model 
cannot be distinguished from the effect of the particular staff, local services, or other factors at the school that are not part of the 
HCZ organizational model. 


4 Studies had to be released or made public in 1996 or later and be obtained by the WWC for review before the drafting of the HCZ 
Promise Academy Charter Schools intervention report. There were no studies released or made public before the first HCZ charter 
school opened in 2004. 


5 Please see the Charter Schools review protocol (version 3.0) for details on the types of interventions that are eligible for review. A 
study of the effectiveness of an individual charter school is eligible to be included in a review of the evidence of the effectiveness of an 
individual charter school, but is not eligible to be included in a review of the evidence of the effectiveness of anamed CMO or charter 
network, like HCZ. The Dobbie and Fryer (2015) as well as the Whitehurst and Croft (2010) studies were conducted in one of HCZ’s 
Promise Academy Charter Schools, so they are not eligible to be included in this review. 
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Glossary of Terms 


Attrition 


Baseline 


Clustering adjustment 


Confounding factor 


Design 


Effect size 
Eligibility 


Equivalence 


Attrition occurs when an outcome variable is not available for all subjects initially assigned to 
the intervention and comparison groups. If a randomized controlled trial (RCT) or regression 
discontinuity design (RDD) study has high levels of attrition, the validity of the study results 
can be called into question. An RCT with high attrition cannot receive the highest rating of 
Meets WWC Group Design Standards without Reservations, but can receive a rating of Meets 
WWC Group Design Standards with Reservations if it establishes baseline equivalence of the 
analytic sample. Similarly, the highest rating an RDD with high attrition can receive is Meets 
WWC RDD Standards with Reservations. 


For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and individuals 
leave the group. If a single-case design does not meet minimum requirements for phases and 
data points within phases, the study cannot receive the highest rating of Meets WWC Pilot 
Single-Case Design Standards without Reservations. 


A point in time before the intervention was implemented in group design research and in 
regression discontinuity design studies. When a study is required to satisfy the baseline 
equivalence requirement, it must be done with characteristics of the analytic sample at 
baseline. In a single-case design experiment, the baseline condition is a period during 
which participants are not receiving the intervention. 


An adjustment to the statistical significance of a finding when the units of assignment and 
analysis differ. When random assignment is carried out at the cluster level, outcomes for 
individual units within the same clusters may be correlated. When the analysis is conducted 
at the individual level rather than the cluster level, there is a mismatch between the unit 

of assignment and the unit of analysis, and this correlation must be accounted for when 
assessing the statistical significance of an impact estimate. If the correlation is not 
accounted for in a mismatched analysis, the study may be too likely to report statistically 
significant findings. To fairly assess an intervention’s effects, in cases where study authors 
have not corrected for the clustering, the WWC applies an adjustment for clustering when 
reporting statistical significance. 


A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 


The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed 
repeatedly within and across different phases that are defined by the presence or absence 
of an intervention (single-case design). Designs eligible for WWC review are randomized 
controlled trials, quasi-experimental designs, regression discontinuity designs, and 
single-case designs. 


The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 


A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 


A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 
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Extent of evidence An indication of how much evidence from group design studies supports the findings in an 
intervention report. The extent of evidence categorization for intervention reports focuses 
on the number and sizes of studies of the intervention in order to give an indication of how 
broadly findings may be applied to different settings. There are two extent of evidence 
categories: small and medium to large. 


e small: includes only one study, or one school, or findings based on a total sample size 
of less than 350 students and 14 classrooms (assuming 25 students in a class) 


e medium to large: includes more than one study, more than one school, and findings 
based on a total sample of at least 350 students or 14 classrooms 


Gain scores The result of subtracting the pretest from the posttest for each individual in the sample. 
Some studies analyze gain scores instead of the unadjusted outcome measure as a method 
of accounting for the baseline measure when estimating the effect of an intervention. The 
WWC reviews and reports findings from analyses of gain scores, but gain scores do not 
satisfy the WWC’s requirement for a statistical adjustment under the baseline equivalence 
requirement. This means that a study that must satisfy the baseline equivalence requirement 
and has baseline differences between 0.05 and 0.25 standard deviations Does Not Meet 
WWC Group Design Standards if the study’s only adjustment for the baseline measure was 
in the construction of the gain score. 


Group design A study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 


Improvement index Along a percentile distribution of individuals, the improvement index represents the gain or 
loss of the average individual due to the intervention. As the average individual starts at the 
50th percentile, the measure ranges from —50 to +50. 


Intervention An educational program, product, practice, or policy aimed at improving student outcomes. 


Intervention report A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an 
intervention, reviews each against design standards, and summarizes the findings of those 
that meet WWC design standards. 


Multiple comparison An adjustment to the statistical significance of results to account for multiple comparisons 
adjustment in a group design study. The WWC uses the Benjamini-Hochberg (BH) correction to adjust 

the statistical significance of results within an outcome domain when study authors perform 
multiple hypothesis tests without adjusting the p-value. The BH correction is used in three 
types of situations: studies that tested multiple outcome measures in the same outcome 
domain with a single comparison group; studies that tested a given outcome measure 
with multiple comparison groups; and studies that tested multiple outcome measures in 
the same outcome domain with multiple comparison groups. Because repeated tests of 
highly correlated constructs will lead to a greater likelihood of mistakenly concluding that 
the impact was different from zero, in all three situations, the WWC uses the BH correction 
to reduce the possibility of making this error. The WWC makes separate adjustments for 
primary and secondary findings. 


Outcome domain A group of closely-related outcomes. A domain is the organizing construct for a set of 
related outcomes through which studies claim effectiveness. 
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Quasi-experimental 
design (QED) 


Randomized controlled 
trial (RCT) 


Rating of Effectiveness 


Regression discontinuity 
design (RDD) 


Single-case design 


Standard deviation 


Statistical significance 


Study rating 


Substantively important 


Systematic review 


A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 


A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 


For group design research, the WWC rates the effectiveness of an intervention in each 
domain based on the quality of the research design and the magnitude, statistical signifi- 
cance, and consistency in findings. For single-case design research, the WWC rates the 
effectiveness of an intervention in each domain based on the quality of the research design 
and the consistency of demonstrated effects. 


A design in which groups are created using a continuous scoring rule. For example, 
students may be assigned to a summer school program if they score below a preset point 
on a standardized test, or schools may be awarded a grant based on their score on an 
application. A regression line or curve is estimated for the intervention group and similarly 
for the comparison group, and an effect occurs if there is a discontinuity in the two 
regression lines at the cutoff. 


A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 


The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 


Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding 
statistically significant if the likelihood that the difference is due to chance is less than 
5% (p < .05). 


The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets WWC Design Standards without Reservations, Meets WWC Design Standards with 
Reservations, or Does Not Meet WWC Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 


A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


A review of existing literature on a topic that is identified and reviewed using explicit 
methods. A WWC systematic review has five steps: 1) developing a review protocol; 

2) searching the literature; 3) reviewing studies, including screening studies for eligibility, 
reviewing the methodological quality of each study, and reporting on high quality studies 
and their findings; 4) combining findings within and across studies; and, 5) summarizing 
the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention Practice Quick Single Study 
Report Guide Review Review 


An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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